{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# FastRNN and FastGRNN in Tensorflow\n",
    "\n",
    "This is a simple notebook that illustrates the usage of Tensorflow implementation of FastRNN and FastGRNN. We are using the USPS dataset. Please refer to `fetch_usps.py` and run it for downloading and cleaning up the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright (c) Microsoft Corporation. All rights reserved.\n",
    "# Licensed under the MIT license.\n",
    "\n",
    "import helpermethods\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import sys\n",
    "import os\n",
    "\n",
    "#Provide the GPU number to be used\n",
    "os.environ['CUDA_VISIBLE_DEVICES'] =''\n",
    "\n",
    "#FastRNN and FastGRNN imports\n",
    "from edgeml_tf.trainer.fastTrainer import FastTrainer\n",
    "from edgeml_tf.graph.rnn import FastGRNNCell\n",
    "from edgeml_tf.graph.rnn import FastRNNCell\n",
    "from edgeml_tf.graph.rnn import UGRNNLRCell\n",
    "from edgeml_tf.graph.rnn import GRULRCell\n",
    "from edgeml_tf.graph.rnn import LSTMLRCell\n",
    "\n",
    "# Fixing seeds for reproducibility\n",
    "tf.set_random_seed(42)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# USPS Data\n",
    "\n",
    "It is assumed that the USPS data has already been downloaded and processed with [fetch_usps.py](fetch_usps.py) and [process_usps.py](process_usps.py), and is present in the `./usps10` subdirectory.\n",
    "\n",
    "Note: Even though usps10 is not a time-series dataset, it can be assumed as, a time-series where each row is coming in at one single time.\n",
    "So the number of timesteps = 16 and inputDims = 16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature Dimension:  256\n",
      "Num classes:  10\n"
     ]
    }
   ],
   "source": [
    "#Loading and Pre-processing dataset for FastCells\n",
    "dataDir = \"usps10\"\n",
    "(dataDimension, numClasses, Xtrain, Ytrain, Xtest, Ytest, mean, std) = helpermethods.preProcessData(dataDir)\n",
    "print(\"Feature Dimension: \", dataDimension)\n",
    "print(\"Num classes: \", numClasses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Parameters\n",
    "\n",
    "FastRNN and FastGRNN work for most of the hyper-parameters with which you could acheive decent accuracies on LSTM/GRU. Over and above that, you can use low-rank, sparsity and quatization to reduce model size upto 45x when compared to LSTM/GRU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "cell = \"FastGRNN\" # Choose between FastGRNN, FastRNN, UGRNN, GRU and LSTM\n",
    "\n",
    "inputDims = 16 #features taken in by RNN in one timestep\n",
    "hiddenDims = 32 #hidden state of RNN\n",
    "\n",
    "totalEpochs = 300\n",
    "batchSize = 100\n",
    "\n",
    "learningRate = 0.01\n",
    "decayStep = 200\n",
    "decayRate = 0.1\n",
    "\n",
    "outFile = None #provide your file, if you need all the logging info in a file\n",
    "\n",
    "#low-rank parameterisation for weight matrices. None => Full Rank\n",
    "wRank = None \n",
    "uRank = None \n",
    "\n",
    "#Sparsity of the weight matrices. x => 100*x % are non-zeros\n",
    "sW = 1.0 \n",
    "sU = 1.0\n",
    "\n",
    "#Non-linearities for the RNN architecture. Can choose from \"tanh, sigmoid, relu, quantTanh, quantSigm\"\n",
    "update_non_linearity = \"tanh\"\n",
    "gate_non_linearity = \"sigmoid\"\n",
    "\n",
    "assert dataDimension % inputDims == 0, \"Infeasible per step input, Timesteps have to be integer\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Placeholders for Data feeding during training and infernece"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = tf.placeholder(\"float\", [None, int(dataDimension / inputDims), inputDims])\n",
    "Y = tf.placeholder(\"float\", [None, numClasses])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating a directory for current model in the datadirectory using timestamp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "currDir = helpermethods.createTimeStampDir(dataDir, cell)\n",
    "helpermethods.dumpCommand(sys.argv, currDir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# FastCell Graph Object\n",
    "\n",
    "Instantiating the FastCell Graph using modular RNN Cells which will be used for training and inference.\n",
    "\n",
    "Note: RNN cells in edgeml.rnn can be used anywhere in place of LSTM/GRU in a plug & play fashion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Create appropriate RNN cell object based on choice\n",
    "if cell == \"FastGRNN\":\n",
    "    FastCell = FastGRNNCell(hiddenDims, gate_non_linearity=gate_non_linearity,\n",
    "                            update_non_linearity=update_non_linearity,\n",
    "                            wRank=wRank, uRank=uRank)\n",
    "elif cell == \"FastRNN\":\n",
    "    FastCell = FastRNNCell(hiddenDims, update_non_linearity=update_non_linearity,\n",
    "                           wRank=wRank, uRank=uRank)\n",
    "elif cell == \"UGRNN\":\n",
    "    FastCell = UGRNNLRCell(hiddenDims, update_non_linearity=update_non_linearity,\n",
    "                           wRank=wRank, uRank=uRank)\n",
    "elif cell == \"GRU\":\n",
    "    FastCell = GRULRCell(hiddenDims, update_non_linearity=update_non_linearity,\n",
    "                         wRank=wRank, uRank=uRank)\n",
    "elif cell == \"LSTM\":\n",
    "    FastCell = LSTMLRCell(hiddenDims, update_non_linearity=update_non_linearity,\n",
    "                          wRank=wRank, uRank=uRank)\n",
    "else:\n",
    "    sys.exit('Exiting: No Such Cell as ' + cell)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# FastCell Trainer Object\n",
    "\n",
    "Instantiating the FastCell Trainer which will be used for 3 phase training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "FastCellTrainer = FastTrainer(FastCell, X, Y, sW=sW, sU=sU, learningRate=learningRate, outFile=outFile)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Session declaration and variable initialization. Interactive Session doesn't clog the entire GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess = tf.InteractiveSession()\n",
    "sess.run(tf.global_variables_initializer())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# FastCell Training Routine\n",
    "\n",
    "The method to to run the 3 phase training, followed by giving out the best early stopping model, accuracy along with saving of the parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch Number: 0\n",
      "\n",
      "******************** Dense Training Phase Started ********************\n",
      "\n",
      "Train Loss: 1.3531070024999854 Train Accuracy: 0.565881378744563\n",
      "Test Loss: 0.8334901 Test Accuracy: 0.7349278\n",
      "\n",
      "Epoch Number: 1\n",
      "Train Loss: 0.5264064224615489 Train Accuracy: 0.8227005854044875\n",
      "Test Loss: 0.52811986 Test Accuracy: 0.83557546\n",
      "\n",
      "Epoch Number: 2\n",
      "Train Loss: 0.3170111432467421 Train Accuracy: 0.8997546287432109\n",
      "Test Loss: 0.41971388 Test Accuracy: 0.87593424\n",
      "\n",
      "Epoch Number: 3\n",
      "Train Loss: 0.22838621382435706 Train Accuracy: 0.9285217539904869\n",
      "Test Loss: 0.37176716 Test Accuracy: 0.8943697\n",
      "\n",
      "Epoch Number: 4\n",
      "Train Loss: 0.17584358977332507 Train Accuracy: 0.9436173479850978\n",
      "Test Loss: 0.3482268 Test Accuracy: 0.9013453\n",
      "\n",
      "Epoch Number: 5\n",
      "Train Loss: 0.1554100387921072 Train Accuracy: 0.9503703141865665\n",
      "Test Loss: 0.36468038 Test Accuracy: 0.8963627\n",
      "\n",
      "Epoch Number: 6\n",
      "Train Loss: 0.13128593576791353 Train Accuracy: 0.9591509887616928\n",
      "Test Loss: 0.36238122 Test Accuracy: 0.9028401\n",
      "\n",
      "Epoch Number: 7\n",
      "Train Loss: 0.11856559077150201 Train Accuracy: 0.9623016780369902\n",
      "Test Loss: 0.37148365 Test Accuracy: 0.9003488\n",
      "\n",
      "Epoch Number: 8\n",
      "Train Loss: 0.11480801381579 Train Accuracy: 0.9623016764039862\n",
      "Test Loss: 0.40140042 Test Accuracy: 0.8938714\n",
      "\n",
      "Epoch Number: 9\n",
      "Train Loss: 0.11065655635440186 Train Accuracy: 0.9653153754260442\n",
      "Test Loss: 0.3517686 Test Accuracy: 0.90981567\n",
      "\n",
      "Epoch Number: 10\n",
      "Train Loss: 0.09199796772676788 Train Accuracy: 0.9716302948455288\n",
      "Test Loss: 0.3499246 Test Accuracy: 0.9147982\n",
      "\n",
      "Epoch Number: 11\n",
      "Train Loss: 0.07985301451017596 Train Accuracy: 0.9762742788824317\n",
      "Test Loss: 0.3625236 Test Accuracy: 0.91529644\n",
      "\n",
      "Epoch Number: 12\n",
      "Train Loss: 0.07171525779397112 Train Accuracy: 0.9787535806224771\n",
      "Test Loss: 0.35705435 Test Accuracy: 0.91429996\n",
      "\n",
      "Epoch Number: 13\n",
      "Train Loss: 0.077431221046064 Train Accuracy: 0.9755893504782899\n",
      "Test Loss: 0.38592914 Test Accuracy: 0.9093174\n",
      "\n",
      "Epoch Number: 14\n",
      "Train Loss: 0.07726132686007513 Train Accuracy: 0.9744799128950459\n",
      "Test Loss: 0.38768652 Test Accuracy: 0.9123069\n",
      "\n",
      "Epoch Number: 15\n",
      "Train Loss: 0.06339540997239416 Train Accuracy: 0.9798494748873253\n",
      "Test Loss: 0.36402556 Test Accuracy: 0.9197808\n",
      "\n",
      "Epoch Number: 16\n",
      "Train Loss: 0.0624726844173282 Train Accuracy: 0.9810823528733972\n",
      "Test Loss: 0.3556986 Test Accuracy: 0.9192825\n",
      "\n",
      "Epoch Number: 17\n",
      "Train Loss: 0.05848091944082551 Train Accuracy: 0.9821376008530186\n",
      "Test Loss: 0.3734596 Test Accuracy: 0.922272\n",
      "\n",
      "Epoch Number: 18\n",
      "Train Loss: 0.06179975296613084 Train Accuracy: 0.9775207050859112\n",
      "Test Loss: 0.37375587 Test Accuracy: 0.9147982\n",
      "\n",
      "Epoch Number: 19\n",
      "Train Loss: 0.060816061236474615 Train Accuracy: 0.980534406557475\n",
      "Test Loss: 0.36386096 Test Accuracy: 0.92077726\n",
      "\n",
      "Epoch Number: 20\n",
      "Train Loss: 0.05517878877126599 Train Accuracy: 0.9829866126792072\n",
      "Test Loss: 0.38278854 Test Accuracy: 0.92077726\n",
      "\n",
      "Epoch Number: 21\n",
      "Train Loss: 0.04950164187036148 Train Accuracy: 0.9835481072125369\n",
      "Test Loss: 0.38189712 Test Accuracy: 0.91878426\n",
      "\n",
      "Epoch Number: 22\n",
      "Train Loss: 0.04603105507893105 Train Accuracy: 0.984219489848777\n",
      "Test Loss: 0.39881724 Test Accuracy: 0.9123069\n",
      "\n",
      "Epoch Number: 23\n",
      "Train Loss: 0.04120528124183519 Train Accuracy: 0.985726339359806\n",
      "Test Loss: 0.41953668 Test Accuracy: 0.91131043\n",
      "\n",
      "Epoch Number: 24\n",
      "Train Loss: 0.04223672329282312 Train Accuracy: 0.9858497748636219\n",
      "Test Loss: 0.37599987 Test Accuracy: 0.9227703\n",
      "\n",
      "Epoch Number: 25\n",
      "Train Loss: 0.044115278812457026 Train Accuracy: 0.9849044190694208\n",
      "Test Loss: 0.39963064 Test Accuracy: 0.92127556\n",
      "\n",
      "Epoch Number: 26\n",
      "Train Loss: 0.060125956299064094 Train Accuracy: 0.9792608863686862\n",
      "Test Loss: 0.39676014 Test Accuracy: 0.91131043\n",
      "\n",
      "Epoch Number: 27\n",
      "Train Loss: 0.058513890084338514 Train Accuracy: 0.9795484101935609\n",
      "Test Loss: 0.3695973 Test Accuracy: 0.9217738\n",
      "\n",
      "Epoch Number: 28\n",
      "Train Loss: 0.04882802803401057 Train Accuracy: 0.9824115707449717\n",
      "Test Loss: 0.4062322 Test Accuracy: 0.9128052\n",
      "\n",
      "Epoch Number: 29\n",
      "Train Loss: 0.04246805129853422 Train Accuracy: 0.9854659160522565\n",
      "Test Loss: 0.36979795 Test Accuracy: 0.92526156\n",
      "\n",
      "Epoch Number: 30\n",
      "Train Loss: 0.05128337493906283 Train Accuracy: 0.9843700242369142\n",
      "Test Loss: 0.4025077 Test Accuracy: 0.9172895\n",
      "\n",
      "Epoch Number: 31\n",
      "Train Loss: 0.04524477290895398 Train Accuracy: 0.9840825028615455\n",
      "Test Loss: 0.36316648 Test Accuracy: 0.9227703\n",
      "\n",
      "Epoch Number: 32\n",
      "Train Loss: 0.04791155387966396 Train Accuracy: 0.9839319660239023\n",
      "Test Loss: 0.38224837 Test Accuracy: 0.9197808\n",
      "\n",
      "Epoch Number: 33\n",
      "Train Loss: 0.04305804770261253 Train Accuracy: 0.98493151713724\n",
      "Test Loss: 0.3597 Test Accuracy: 0.9217738\n",
      "\n",
      "Epoch Number: 34\n",
      "Train Loss: 0.03439056758819888 Train Accuracy: 0.9891509944445467\n",
      "Test Loss: 0.36144 Test Accuracy: 0.92326856\n",
      "\n",
      "Epoch Number: 35\n",
      "Train Loss: 0.025825574640057063 Train Accuracy: 0.9935481017583037\n",
      "Test Loss: 0.3576532 Test Accuracy: 0.9287494\n",
      "\n",
      "Epoch Number: 36\n",
      "Train Loss: 0.020732127933775726 Train Accuracy: 0.9947809772948696\n",
      "Test Loss: 0.3529356 Test Accuracy: 0.92825115\n",
      "\n",
      "Epoch Number: 37\n",
      "Train Loss: 0.02256068464189972 Train Accuracy: 0.9938356215006685\n",
      "Test Loss: 0.3675873 Test Accuracy: 0.93223715\n",
      "\n",
      "Epoch Number: 38\n",
      "Train Loss: 0.04096006839025817 Train Accuracy: 0.9857398875772136\n",
      "Test Loss: 0.36569017 Test Accuracy: 0.9267564\n",
      "\n",
      "Epoch Number: 39\n",
      "Train Loss: 0.04014190110339694 Train Accuracy: 0.9867123389897281\n",
      "Test Loss: 0.34677818 Test Accuracy: 0.9262581\n",
      "\n",
      "Epoch Number: 40\n",
      "Train Loss: 0.031071233378136404 Train Accuracy: 0.9899864605028336\n",
      "Test Loss: 0.363686 Test Accuracy: 0.92775285\n",
      "\n",
      "Epoch Number: 41\n",
      "Train Loss: 0.02729316997303538 Train Accuracy: 0.9908219265611204\n",
      "Test Loss: 0.35555694 Test Accuracy: 0.9312407\n",
      "\n",
      "Epoch Number: 42\n",
      "Train Loss: 0.021803765849542026 Train Accuracy: 0.992191786635412\n",
      "Test Loss: 0.35095477 Test Accuracy: 0.93223715\n",
      "\n",
      "Epoch Number: 43\n",
      "Train Loss: 0.04842862480460373 Train Accuracy: 0.9833975695583919\n",
      "Test Loss: 0.42905322 Test Accuracy: 0.91679126\n",
      "\n",
      "Epoch Number: 44\n",
      "Train Loss: 0.04453416636264692 Train Accuracy: 0.9834111210418074\n",
      "Test Loss: 0.406023 Test Accuracy: 0.920279\n",
      "\n",
      "Epoch Number: 45\n",
      "Train Loss: 0.038877726283740914 Train Accuracy: 0.9870962010671015\n",
      "Test Loss: 0.39293337 Test Accuracy: 0.91878426\n",
      "\n",
      "Epoch Number: 46\n",
      "Train Loss: 0.034626684416315126 Train Accuracy: 0.9884796118083066\n",
      "Test Loss: 0.36277694 Test Accuracy: 0.9237668\n",
      "\n",
      "Epoch Number: 47\n",
      "Train Loss: 0.02302065390889367 Train Accuracy: 0.9934111139545702\n",
      "Test Loss: 0.38474992 Test Accuracy: 0.9247633\n",
      "\n",
      "Epoch Number: 48\n",
      "Train Loss: 0.023432086993723292 Train Accuracy: 0.9943564705652733\n",
      "Test Loss: 0.370669 Test Accuracy: 0.9237668\n",
      "\n",
      "Epoch Number: 49\n",
      "Train Loss: 0.024380253930097726 Train Accuracy: 0.9921782384180042\n",
      "Test Loss: 0.40583202 Test Accuracy: 0.9227703\n",
      "\n",
      "Epoch Number: 50\n",
      "Train Loss: 0.023330659918129854 Train Accuracy: 0.9926027467806046\n",
      "Test Loss: 0.4097609 Test Accuracy: 0.92575985\n",
      "\n",
      "Epoch Number: 51\n",
      "Train Loss: 0.018314683679108545 Train Accuracy: 0.9943835661835867\n",
      "Test Loss: 0.38972235 Test Accuracy: 0.9342302\n",
      "\n",
      "Epoch Number: 52\n",
      "Train Loss: 0.029633181783600315 Train Accuracy: 0.9905344043692498\n",
      "Test Loss: 0.37864792 Test Accuracy: 0.9247633\n",
      "\n",
      "Epoch Number: 53\n",
      "Train Loss: 0.030011002424058235 Train Accuracy: 0.9905479509536534\n",
      "Test Loss: 0.3964535 Test Accuracy: 0.9192825\n",
      "\n",
      "Epoch Number: 54\n",
      "Train Loss: 0.03564942483343694 Train Accuracy: 0.9888499256682722\n",
      "Test Loss: 0.38546467 Test Accuracy: 0.92326856\n",
      "\n",
      "Epoch Number: 55\n",
      "Train Loss: 0.0320119748230105 Train Accuracy: 0.9893015280161819\n",
      "Test Loss: 0.41079342 Test Accuracy: 0.91679126\n",
      "\n",
      "Epoch Number: 56\n",
      "Train Loss: 0.027233783602204225 Train Accuracy: 0.9919042677095492\n",
      "Test Loss: 0.40080228 Test Accuracy: 0.9217738\n",
      "\n",
      "Epoch Number: 57\n",
      "Train Loss: 0.0170260006386455 Train Accuracy: 0.9949044152481915\n",
      "Test Loss: 0.42503983 Test Accuracy: 0.9292476\n",
      "\n",
      "Epoch Number: 58\n",
      "Train Loss: 0.020110745480513736 Train Accuracy: 0.9946575393415478\n",
      "Test Loss: 0.38848647 Test Accuracy: 0.9217738\n",
      "\n",
      "Epoch Number: 59\n",
      "Train Loss: 0.015590530762780611 Train Accuracy: 0.9949179634655991\n",
      "Test Loss: 0.4031199 Test Accuracy: 0.92775285\n",
      "\n",
      "Epoch Number: 60\n",
      "Train Loss: 0.022963548624530844 Train Accuracy: 0.992863170088154\n",
      "Test Loss: 0.42644864 Test Accuracy: 0.9197808\n",
      "\n",
      "Epoch Number: 61\n",
      "Train Loss: 0.024166807283532536 Train Accuracy: 0.9914933091973606\n",
      "Test Loss: 0.4117787 Test Accuracy: 0.9247633\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch Number: 62\n",
      "Train Loss: 0.01902851595364715 Train Accuracy: 0.9946575393415478\n",
      "Test Loss: 0.43569365 Test Accuracy: 0.918286\n",
      "\n",
      "Epoch Number: 63\n",
      "Train Loss: 0.022098659849502402 Train Accuracy: 0.9919178142939529\n",
      "Test Loss: 0.4453173 Test Accuracy: 0.92575985\n",
      "\n",
      "Epoch Number: 64\n",
      "Train Loss: 0.02353779313932747 Train Accuracy: 0.9930001562588835\n",
      "Test Loss: 0.43414015 Test Accuracy: 0.91429996\n",
      "\n",
      "Epoch Number: 65\n",
      "Train Loss: 0.016468530626048986 Train Accuracy: 0.9947809764783676\n",
      "Test Loss: 0.43052217 Test Accuracy: 0.9217738\n",
      "\n",
      "Epoch Number: 66\n",
      "Train Loss: 0.016379667304086257 Train Accuracy: 0.9958904148781136\n",
      "Test Loss: 0.4004999 Test Accuracy: 0.92825115\n",
      "\n",
      "Epoch Number: 67\n",
      "Train Loss: 0.012232361819072026 Train Accuracy: 0.9971232904146795\n",
      "Test Loss: 0.40298688 Test Accuracy: 0.93273544\n",
      "\n",
      "Epoch Number: 68\n",
      "Train Loss: 0.008708359920403806 Train Accuracy: 0.998493152121975\n",
      "Test Loss: 0.42018083 Test Accuracy: 0.9272546\n",
      "\n",
      "Epoch Number: 69\n",
      "Train Loss: 0.009453040786081134 Train Accuracy: 0.9979452074390568\n",
      "Test Loss: 0.42367473 Test Accuracy: 0.9287494\n",
      "\n",
      "Epoch Number: 70\n",
      "Train Loss: 0.02633900548393634 Train Accuracy: 0.9916031989332748\n",
      "Test Loss: 0.4314282 Test Accuracy: 0.91778773\n",
      "\n",
      "Epoch Number: 71\n",
      "Train Loss: 0.05996186181277751 Train Accuracy: 0.9832605866536702\n",
      "Test Loss: 0.40858173 Test Accuracy: 0.9227703\n",
      "\n",
      "Epoch Number: 72\n",
      "Train Loss: 0.03984937108479032 Train Accuracy: 0.9866987883228145\n",
      "Test Loss: 0.41035435 Test Accuracy: 0.9247633\n",
      "\n",
      "Epoch Number: 73\n",
      "Train Loss: 0.024671344705283232 Train Accuracy: 0.991356323026631\n",
      "Test Loss: 0.42347214 Test Accuracy: 0.92575985\n",
      "\n",
      "Epoch Number: 74\n",
      "Train Loss: 0.0261542204694108 Train Accuracy: 0.9923287736226435\n",
      "Test Loss: 0.40737543 Test Accuracy: 0.92127556\n",
      "\n",
      "Epoch Number: 75\n",
      "Train Loss: 0.021734511994833304 Train Accuracy: 0.9932741294168446\n",
      "Test Loss: 0.38865966 Test Accuracy: 0.93024415\n",
      "\n",
      "Epoch Number: 76\n",
      "Train Loss: 0.017603178070028862 Train Accuracy: 0.9942330326119514\n",
      "Test Loss: 0.41292053 Test Accuracy: 0.9272546\n",
      "\n",
      "Epoch Number: 77\n",
      "Train Loss: 0.01774944902368987 Train Accuracy: 0.9949179634655991\n",
      "Test Loss: 0.3975856 Test Accuracy: 0.9287494\n",
      "\n",
      "Epoch Number: 78\n",
      "Train Loss: 0.026556726039565895 Train Accuracy: 0.9926027451476006\n",
      "Test Loss: 0.37759724 Test Accuracy: 0.9262581\n",
      "\n",
      "Epoch Number: 79\n",
      "Train Loss: 0.03763930009971436 Train Accuracy: 0.9888905711369972\n",
      "Test Loss: 0.44735578 Test Accuracy: 0.9172895\n",
      "\n",
      "Epoch Number: 80\n",
      "Train Loss: 0.029481777800119496 Train Accuracy: 0.9901369957074727\n",
      "Test Loss: 0.41876832 Test Accuracy: 0.9217738\n",
      "\n",
      "Epoch Number: 81\n",
      "Train Loss: 0.02179017917999411 Train Accuracy: 0.9924522115759653\n",
      "Test Loss: 0.41835007 Test Accuracy: 0.920279\n",
      "\n",
      "Epoch Number: 82\n",
      "Train Loss: 0.0234184642127007 Train Accuracy: 0.9921646918336006\n",
      "Test Loss: 0.41416502 Test Accuracy: 0.9272546\n",
      "\n",
      "Epoch Number: 83\n",
      "Train Loss: 0.02082834580166852 Train Accuracy: 0.993260580382935\n",
      "Test Loss: 0.4422068 Test Accuracy: 0.9172895\n",
      "\n",
      "Epoch Number: 84\n",
      "Train Loss: 0.022050149352465156 Train Accuracy: 0.9939726084879\n",
      "Test Loss: 0.3987477 Test Accuracy: 0.92825115\n",
      "\n",
      "Epoch Number: 85\n",
      "Train Loss: 0.026048276352549405 Train Accuracy: 0.9927261847339265\n",
      "Test Loss: 0.38845396 Test Accuracy: 0.9272546\n",
      "\n",
      "Epoch Number: 86\n",
      "Train Loss: 0.01715031830029409 Train Accuracy: 0.9952054840244658\n",
      "Test Loss: 0.3792558 Test Accuracy: 0.9267564\n",
      "\n",
      "Epoch Number: 87\n",
      "Train Loss: 0.014544817494636732 Train Accuracy: 0.9964248113436242\n",
      "Test Loss: 0.41980278 Test Accuracy: 0.9242651\n",
      "\n",
      "Epoch Number: 88\n",
      "Train Loss: 0.006491333439193462 Train Accuracy: 0.9987671244634341\n",
      "Test Loss: 0.39655796 Test Accuracy: 0.9317389\n",
      "\n",
      "Epoch Number: 89\n",
      "Train Loss: 0.004307456604007325 Train Accuracy: 0.9993150691463523\n",
      "Test Loss: 0.40233433 Test Accuracy: 0.92825115\n",
      "\n",
      "Epoch Number: 90\n",
      "Train Loss: 0.0027800448436596215 Train Accuracy: 0.9997260276585409\n",
      "Test Loss: 0.403938 Test Accuracy: 0.9287494\n",
      "\n",
      "Epoch Number: 91\n",
      "Train Loss: 0.002242555237002033 Train Accuracy: 0.9995890414878114\n",
      "Test Loss: 0.4070075 Test Accuracy: 0.9307424\n",
      "\n",
      "Epoch Number: 92\n",
      "Train Loss: 0.0022119151863703277 Train Accuracy: 0.9995890414878114\n",
      "Test Loss: 0.41036773 Test Accuracy: 0.9307424\n",
      "\n",
      "Epoch Number: 93\n",
      "Train Loss: 0.001824945211809205 Train Accuracy: 0.9997260276585409\n",
      "Test Loss: 0.41361076 Test Accuracy: 0.9317389\n",
      "\n",
      "Epoch Number: 94\n",
      "Train Loss: 0.001808816738895536 Train Accuracy: 0.9997260276585409\n",
      "Test Loss: 0.41818038 Test Accuracy: 0.93223715\n",
      "\n",
      "Epoch Number: 95\n",
      "Train Loss: 0.0015898832340871482 Train Accuracy: 0.9997260276585409\n",
      "Test Loss: 0.4229942 Test Accuracy: 0.9312407\n",
      "\n",
      "Epoch Number: 96\n",
      "Train Loss: 0.001751650427787067 Train Accuracy: 0.9997260276585409\n",
      "Test Loss: 0.42656386 Test Accuracy: 0.9332337\n",
      "\n",
      "Epoch Number: 97\n",
      "Train Loss: 0.0015788370674023125 Train Accuracy: 0.9997260276585409\n",
      "Test Loss: 0.43016008 Test Accuracy: 0.93223715\n",
      "\n",
      "Epoch Number: 98\n",
      "Train Loss: 0.0016806908206988688 Train Accuracy: 0.9997260276585409\n",
      "Test Loss: 0.43488127 Test Accuracy: 0.9297459\n",
      "\n",
      "Epoch Number: 99\n",
      "Train Loss: 0.0015810940553009996 Train Accuracy: 0.9997260276585409\n",
      "Test Loss: 0.43672302 Test Accuracy: 0.9307424\n",
      "\n",
      "Epoch Number: 100\n",
      "Train Loss: 0.001932646346052037 Train Accuracy: 0.9995890414878114\n",
      "Test Loss: 0.4472658 Test Accuracy: 0.9292476\n",
      "\n",
      "Epoch Number: 101\n",
      "Train Loss: 0.03748205324996914 Train Accuracy: 0.9887535817002597\n",
      "Test Loss: 0.44853446 Test Accuracy: 0.9147982\n",
      "\n",
      "Epoch Number: 102\n",
      "Train Loss: 0.11106032654898215 Train Accuracy: 0.9678774721001926\n",
      "Test Loss: 0.36246482 Test Accuracy: 0.9227703\n",
      "\n",
      "Epoch Number: 103\n",
      "Train Loss: 0.05241849743569755 Train Accuracy: 0.9829866134957091\n",
      "Test Loss: 0.3570766 Test Accuracy: 0.92127556\n",
      "\n",
      "Epoch Number: 104\n",
      "Train Loss: 0.028517094278095723 Train Accuracy: 0.9917672799058157\n",
      "Test Loss: 0.37065578 Test Accuracy: 0.9292476\n",
      "\n",
      "Epoch Number: 105\n",
      "Train Loss: 0.017652338973488915 Train Accuracy: 0.9942330326119514\n",
      "Test Loss: 0.37107578 Test Accuracy: 0.93223715\n",
      "\n",
      "Epoch Number: 106\n",
      "Train Loss: 0.01568616884250245 Train Accuracy: 0.9952054840244658\n",
      "Test Loss: 0.35663217 Test Accuracy: 0.9367215\n",
      "\n",
      "Epoch Number: 107\n",
      "Train Loss: 0.017838217169748084 Train Accuracy: 0.9951919358070582\n",
      "Test Loss: 0.41287652 Test Accuracy: 0.9247633\n",
      "\n",
      "Epoch Number: 108\n",
      "Train Loss: 0.010033470020734717 Train Accuracy: 0.9976712350975977\n",
      "Test Loss: 0.4009026 Test Accuracy: 0.92775285\n",
      "\n",
      "Epoch Number: 109\n",
      "Train Loss: 0.008324489026961925 Train Accuracy: 0.9980686453923787\n",
      "Test Loss: 0.4033402 Test Accuracy: 0.9297459\n",
      "\n",
      "Epoch Number: 110\n",
      "Train Loss: 0.00771069963668371 Train Accuracy: 0.9986165900752969\n",
      "Test Loss: 0.41503954 Test Accuracy: 0.93223715\n",
      "\n",
      "Epoch Number: 111\n",
      "Train Loss: 0.017172634716413608 Train Accuracy: 0.9951919358070582\n",
      "Test Loss: 0.43192545 Test Accuracy: 0.9237668\n",
      "\n",
      "Epoch Number: 112\n",
      "Train Loss: 0.03227482749984842 Train Accuracy: 0.9877946801381569\n",
      "Test Loss: 0.44748837 Test Accuracy: 0.91579473\n",
      "\n",
      "Epoch Number: 113\n",
      "Train Loss: 0.03210181032135215 Train Accuracy: 0.9895890493915506\n",
      "Test Loss: 0.4282059 Test Accuracy: 0.92127556\n",
      "\n",
      "Epoch Number: 114\n",
      "Train Loss: 0.01357129268182365 Train Accuracy: 0.9964383595610318\n",
      "Test Loss: 0.394982 Test Accuracy: 0.92775285\n",
      "\n",
      "Epoch Number: 115\n",
      "Train Loss: 0.019958787539508194 Train Accuracy: 0.9939455112365827\n",
      "Test Loss: 0.44919127 Test Accuracy: 0.92326856\n",
      "\n",
      "Epoch Number: 116\n",
      "Train Loss: 0.01951833989204877 Train Accuracy: 0.9941095938421276\n",
      "Test Loss: 0.39846456 Test Accuracy: 0.9262581\n",
      "\n",
      "Epoch Number: 117\n",
      "Train Loss: 0.013109919135873397 Train Accuracy: 0.9961643872195727\n",
      "Test Loss: 0.3964593 Test Accuracy: 0.93273544\n",
      "\n",
      "Epoch Number: 118\n",
      "Train Loss: 0.008196171877063708 Train Accuracy: 0.9978082212683272\n",
      "Test Loss: 0.39881837 Test Accuracy: 0.93472844\n",
      "\n",
      "Epoch Number: 119\n",
      "Train Loss: 0.0053880620705544285 Train Accuracy: 0.9991780829756227\n",
      "Test Loss: 0.3994898 Test Accuracy: 0.9352267\n",
      "\n",
      "Epoch Number: 120\n",
      "Train Loss: 0.004016872985792436 Train Accuracy: 0.9990410968048932\n",
      "Test Loss: 0.40483266 Test Accuracy: 0.93821627\n",
      "\n",
      "Epoch Number: 121\n",
      "Train Loss: 0.003189845641752807 Train Accuracy: 0.9994520553170818\n",
      "Test Loss: 0.4129637 Test Accuracy: 0.9317389\n",
      "\n",
      "Epoch Number: 122\n",
      "Train Loss: 0.0019790745577105157 Train Accuracy: 0.9997260276585409\n",
      "Test Loss: 0.41090903 Test Accuracy: 0.935725\n",
      "\n",
      "Epoch Number: 123\n",
      "Train Loss: 0.001803544777754873 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.41632676 Test Accuracy: 0.9362232\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch Number: 124\n",
      "Train Loss: 0.0015932139348516189 Train Accuracy: 0.9997260276585409\n",
      "Test Loss: 0.4208521 Test Accuracy: 0.9342302\n",
      "\n",
      "Epoch Number: 125\n",
      "Train Loss: 0.0016064488660697252 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.4273708 Test Accuracy: 0.93721974\n",
      "\n",
      "Epoch Number: 126\n",
      "Train Loss: 0.0015048502509048438 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.43023735 Test Accuracy: 0.9337319\n",
      "\n",
      "Epoch Number: 127\n",
      "Train Loss: 0.0014419755101050824 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.4389877 Test Accuracy: 0.9362232\n",
      "\n",
      "Epoch Number: 128\n",
      "Train Loss: 0.0013684726919028398 Train Accuracy: 0.9997260276585409\n",
      "Test Loss: 0.44143116 Test Accuracy: 0.9342302\n",
      "\n",
      "Epoch Number: 129\n",
      "Train Loss: 0.0013124902181690943 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.44684827 Test Accuracy: 0.93721974\n",
      "\n",
      "Epoch Number: 130\n",
      "Train Loss: 0.001271455863264249 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.44386175 Test Accuracy: 0.9362232\n",
      "\n",
      "Epoch Number: 131\n",
      "Train Loss: 0.0013829727382247642 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.45779392 Test Accuracy: 0.9352267\n",
      "\n",
      "Epoch Number: 132\n",
      "Train Loss: 0.0019963769629288285 Train Accuracy: 0.9995890414878114\n",
      "Test Loss: 0.45586306 Test Accuracy: 0.9352267\n",
      "\n",
      "Epoch Number: 133\n",
      "Train Loss: 0.0016972724874966382 Train Accuracy: 0.9997260276585409\n",
      "Test Loss: 0.4626169 Test Accuracy: 0.935725\n",
      "\n",
      "Epoch Number: 134\n",
      "Train Loss: 0.08464051221679954 Train Accuracy: 0.9767951271305345\n",
      "Test Loss: 0.5116203 Test Accuracy: 0.89287496\n",
      "\n",
      "Epoch Number: 135\n",
      "Train Loss: 0.06996250499282287 Train Accuracy: 0.9749044163586342\n",
      "Test Loss: 0.40689448 Test Accuracy: 0.9242651\n",
      "\n",
      "Epoch Number: 136\n",
      "Train Loss: 0.03188256850970067 Train Accuracy: 0.9891509952610487\n",
      "Test Loss: 0.38704544 Test Accuracy: 0.9262581\n",
      "\n",
      "Epoch Number: 137\n",
      "Train Loss: 0.024611471771032945 Train Accuracy: 0.9905479534031594\n",
      "Test Loss: 0.37838554 Test Accuracy: 0.92526156\n",
      "\n",
      "Epoch Number: 138\n",
      "Train Loss: 0.010400510262315199 Train Accuracy: 0.9975342489268682\n",
      "Test Loss: 0.37540606 Test Accuracy: 0.9367215\n",
      "\n",
      "Epoch Number: 139\n",
      "Train Loss: 0.01360704386503155 Train Accuracy: 0.9961643864030707\n",
      "Test Loss: 0.3835624 Test Accuracy: 0.9332337\n",
      "\n",
      "Epoch Number: 140\n",
      "Train Loss: 0.011857587645589437 Train Accuracy: 0.9971097421972719\n",
      "Test Loss: 0.38600543 Test Accuracy: 0.9337319\n",
      "\n",
      "Epoch Number: 141\n",
      "Train Loss: 0.009779149474263548 Train Accuracy: 0.9979452074390568\n",
      "Test Loss: 0.3920699 Test Accuracy: 0.9292476\n",
      "\n",
      "Epoch Number: 142\n",
      "Train Loss: 0.011075850638356826 Train Accuracy: 0.9968493180732204\n",
      "Test Loss: 0.39335945 Test Accuracy: 0.9337319\n",
      "\n",
      "Epoch Number: 143\n",
      "Train Loss: 0.007224232131018852 Train Accuracy: 0.9982191797805159\n",
      "Test Loss: 0.39289063 Test Accuracy: 0.93721974\n",
      "\n",
      "Epoch Number: 144\n",
      "Train Loss: 0.00687252213119542 Train Accuracy: 0.9986165900752969\n",
      "Test Loss: 0.41478387 Test Accuracy: 0.9342302\n",
      "\n",
      "Epoch Number: 145\n",
      "Train Loss: 0.0036116211602547246 Train Accuracy: 0.9995890414878114\n",
      "Test Loss: 0.38912663 Test Accuracy: 0.9412058\n",
      "\n",
      "Epoch Number: 146\n",
      "Train Loss: 0.002582093849076494 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.38982612 Test Accuracy: 0.93921274\n",
      "\n",
      "Epoch Number: 147\n",
      "Train Loss: 0.0020956868141943793 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.3942846 Test Accuracy: 0.9407075\n",
      "\n",
      "Epoch Number: 148\n",
      "Train Loss: 0.0018172568598943954 Train Accuracy: 0.9997260276585409\n",
      "Test Loss: 0.40042797 Test Accuracy: 0.93971103\n",
      "\n",
      "Epoch Number: 149\n",
      "Train Loss: 0.004224563434110852 Train Accuracy: 0.9993150691463523\n",
      "Test Loss: 0.41389707 Test Accuracy: 0.9387145\n",
      "\n",
      "Epoch Number: 150\n",
      "Train Loss: 0.0033937980600693327 Train Accuracy: 0.9991780829756227\n",
      "Test Loss: 0.4309551 Test Accuracy: 0.9352267\n",
      "\n",
      "Epoch Number: 151\n",
      "Train Loss: 0.016683471513902513 Train Accuracy: 0.995753428707384\n",
      "Test Loss: 0.5002462 Test Accuracy: 0.92326856\n",
      "\n",
      "Epoch Number: 152\n",
      "Train Loss: 0.07921012418587016 Train Accuracy: 0.9773566224803664\n",
      "Test Loss: 0.49312237 Test Accuracy: 0.91380167\n",
      "\n",
      "Epoch Number: 153\n",
      "Train Loss: 0.06044871790321824 Train Accuracy: 0.9809589198190872\n",
      "Test Loss: 0.39408478 Test Accuracy: 0.918286\n",
      "\n",
      "Epoch Number: 154\n",
      "Train Loss: 0.03367851439015047 Train Accuracy: 0.9889041193544048\n",
      "Test Loss: 0.37989596 Test Accuracy: 0.9307424\n",
      "\n",
      "Epoch Number: 155\n",
      "Train Loss: 0.017698209322925196 Train Accuracy: 0.9939319638356771\n",
      "Test Loss: 0.37573016 Test Accuracy: 0.92775285\n",
      "\n",
      "Epoch Number: 156\n",
      "Train Loss: 0.010081476129253383 Train Accuracy: 0.9975342489268682\n",
      "Test Loss: 0.38967404 Test Accuracy: 0.9332337\n",
      "\n",
      "Epoch Number: 157\n",
      "Train Loss: 0.00447188632057227 Train Accuracy: 0.9995890414878114\n",
      "Test Loss: 0.38149583 Test Accuracy: 0.9332337\n",
      "\n",
      "Epoch Number: 158\n",
      "Train Loss: 0.0025207580036120105 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.38982794 Test Accuracy: 0.935725\n",
      "\n",
      "Epoch Number: 159\n",
      "Train Loss: 0.0020615914665092394 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.38763282 Test Accuracy: 0.9362232\n",
      "\n",
      "Epoch Number: 160\n",
      "Train Loss: 0.0017625982677787286 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.39443132 Test Accuracy: 0.93821627\n",
      "\n",
      "Epoch Number: 161\n",
      "Train Loss: 0.0015912198259061432 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.39672822 Test Accuracy: 0.935725\n",
      "\n",
      "Epoch Number: 162\n",
      "Train Loss: 0.0015430196065994693 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.40416223 Test Accuracy: 0.9362232\n",
      "\n",
      "Epoch Number: 163\n",
      "Train Loss: 0.0014482194389143245 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.40405902 Test Accuracy: 0.93721974\n",
      "\n",
      "Epoch Number: 164\n",
      "Train Loss: 0.001342231735877361 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.41373003 Test Accuracy: 0.9342302\n",
      "\n",
      "Epoch Number: 165\n",
      "Train Loss: 0.001386067051797697 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.41112974 Test Accuracy: 0.9387145\n",
      "\n",
      "Epoch Number: 166\n",
      "Train Loss: 0.001193268498392259 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.42294186 Test Accuracy: 0.9332337\n",
      "\n",
      "Epoch Number: 167\n",
      "Train Loss: 0.0012895042981317727 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.41742194 Test Accuracy: 0.9387145\n",
      "\n",
      "Epoch Number: 168\n",
      "Train Loss: 0.001108311818077784 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.4331248 Test Accuracy: 0.9332337\n",
      "\n",
      "Epoch Number: 169\n",
      "Train Loss: 0.001382393570304274 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.42247245 Test Accuracy: 0.93971103\n",
      "\n",
      "Epoch Number: 170\n",
      "Train Loss: 0.0012338795040954184 Train Accuracy: 0.9995890414878114\n",
      "Test Loss: 0.45070615 Test Accuracy: 0.93472844\n",
      "\n",
      "Epoch Number: 171\n",
      "Train Loss: 0.002104504165289069 Train Accuracy: 0.9995890414878114\n",
      "Test Loss: 0.4341877 Test Accuracy: 0.93921274\n",
      "\n",
      "Epoch Number: 172\n",
      "Train Loss: 0.004313505504460534 Train Accuracy: 0.9984796039045674\n",
      "Test Loss: 0.46030065 Test Accuracy: 0.9297459\n",
      "\n",
      "Epoch Number: 173\n",
      "Train Loss: 0.093728030188175 Train Accuracy: 0.9727126351774555\n",
      "Test Loss: 0.42030093 Test Accuracy: 0.91629297\n",
      "\n",
      "Epoch Number: 174\n",
      "Train Loss: 0.05609176247635831 Train Accuracy: 0.9824522162136966\n",
      "Test Loss: 0.41325808 Test Accuracy: 0.92127556\n",
      "\n",
      "Epoch Number: 175\n",
      "Train Loss: 0.02312381442338037 Train Accuracy: 0.9921782392345063\n",
      "Test Loss: 0.4045683 Test Accuracy: 0.9242651\n",
      "\n",
      "Epoch Number: 176\n",
      "Train Loss: 0.014690037730647481 Train Accuracy: 0.9953424693786934\n",
      "Test Loss: 0.3937127 Test Accuracy: 0.9307424\n",
      "\n",
      "Epoch Number: 177\n",
      "Train Loss: 0.00893136704809428 Train Accuracy: 0.9982191797805159\n",
      "Test Loss: 0.39721966 Test Accuracy: 0.92775285\n",
      "\n",
      "Epoch Number: 178\n",
      "Train Loss: 0.007236258619168314 Train Accuracy: 0.9989041106341636\n",
      "Test Loss: 0.38853252 Test Accuracy: 0.92775285\n",
      "\n",
      "Epoch Number: 179\n",
      "Train Loss: 0.004083093933518721 Train Accuracy: 0.9995890414878114\n",
      "Test Loss: 0.37919173 Test Accuracy: 0.9312407\n",
      "\n",
      "Epoch Number: 180\n",
      "Train Loss: 0.002494418898705801 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.39079925 Test Accuracy: 0.935725\n",
      "\n",
      "Epoch Number: 181\n",
      "Train Loss: 0.001869901260624206 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.3961389 Test Accuracy: 0.9312407\n",
      "\n",
      "Epoch Number: 182\n",
      "Train Loss: 0.0017469667874225607 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.4000474 Test Accuracy: 0.93721974\n",
      "\n",
      "Epoch Number: 183\n",
      "Train Loss: 0.0012739899573043908 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.40910247 Test Accuracy: 0.93273544\n",
      "\n",
      "Epoch Number: 184\n",
      "Train Loss: 0.0013601894353672684 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.41040978 Test Accuracy: 0.9362232\n",
      "\n",
      "Epoch Number: 185\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Loss: 0.0011997123000495875 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.41469583 Test Accuracy: 0.935725\n",
      "\n",
      "Epoch Number: 186\n",
      "Train Loss: 0.0014707065065397741 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.41498712 Test Accuracy: 0.94020927\n",
      "\n",
      "Epoch Number: 187\n",
      "Train Loss: 0.002836034407166203 Train Accuracy: 0.9993150691463523\n",
      "Test Loss: 0.45857498 Test Accuracy: 0.9247633\n",
      "\n",
      "Epoch Number: 188\n",
      "Train Loss: 0.04335543119080671 Train Accuracy: 0.9872467313727288\n",
      "Test Loss: 0.4854505 Test Accuracy: 0.91081214\n",
      "\n",
      "Epoch Number: 189\n",
      "Train Loss: 0.0711721541576904 Train Accuracy: 0.9769863102534045\n",
      "Test Loss: 0.38205332 Test Accuracy: 0.92575985\n",
      "\n",
      "Epoch Number: 190\n",
      "Train Loss: 0.028694037625515093 Train Accuracy: 0.9897260347457781\n",
      "Test Loss: 0.3982458 Test Accuracy: 0.9217738\n",
      "\n",
      "Epoch Number: 191\n",
      "Train Loss: 0.020338214381298125 Train Accuracy: 0.9945205531708182\n",
      "Test Loss: 0.39259014 Test Accuracy: 0.9317389\n",
      "\n",
      "Epoch Number: 192\n",
      "Train Loss: 0.015414321870058265 Train Accuracy: 0.995753428707384\n",
      "Test Loss: 0.44486946 Test Accuracy: 0.9247633\n",
      "\n",
      "Epoch Number: 193\n",
      "Train Loss: 0.018497141398655326 Train Accuracy: 0.9928767191220637\n",
      "Test Loss: 0.39597595 Test Accuracy: 0.9242651\n",
      "\n",
      "Epoch Number: 194\n",
      "Train Loss: 0.015433995104203485 Train Accuracy: 0.9949315116830069\n",
      "Test Loss: 0.38098228 Test Accuracy: 0.9332337\n",
      "\n",
      "Epoch Number: 195\n",
      "Train Loss: 0.006958263319712101 Train Accuracy: 0.9980821936097863\n",
      "Test Loss: 0.3999225 Test Accuracy: 0.93273544\n",
      "\n",
      "Epoch Number: 196\n",
      "Train Loss: 0.005096633062213149 Train Accuracy: 0.9991780829756227\n",
      "Test Loss: 0.37073624 Test Accuracy: 0.937718\n",
      "\n",
      "Epoch Number: 197\n",
      "Train Loss: 0.00343738832432866 Train Accuracy: 0.9995890414878114\n",
      "Test Loss: 0.37505808 Test Accuracy: 0.9332337\n",
      "\n",
      "Epoch Number: 198\n",
      "Train Loss: 0.0018556024091818996 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.37871873 Test Accuracy: 0.9342302\n",
      "\n",
      "Epoch Number: 199\n",
      "Train Loss: 0.0015186609035319559 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.3814098 Test Accuracy: 0.935725\n",
      "\n",
      "Epoch Number: 200\n",
      "Train Loss: 0.0010581495521800619 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.38004857 Test Accuracy: 0.9367215\n",
      "\n",
      "Epoch Number: 201\n",
      "Train Loss: 0.0009942382828440925 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.37956578 Test Accuracy: 0.93721974\n",
      "\n",
      "Epoch Number: 202\n",
      "Train Loss: 0.0009491439842561592 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.37938344 Test Accuracy: 0.9367215\n",
      "\n",
      "Epoch Number: 203\n",
      "Train Loss: 0.0009144399371333038 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.37940732 Test Accuracy: 0.93721974\n",
      "\n",
      "Epoch Number: 204\n",
      "Train Loss: 0.0008879157705376027 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.37957668 Test Accuracy: 0.9367215\n",
      "\n",
      "Epoch Number: 205\n",
      "Train Loss: 0.0008668735051648819 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.3798471 Test Accuracy: 0.9362232\n",
      "\n",
      "Epoch Number: 206\n",
      "Train Loss: 0.0008491342837447046 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.38018626 Test Accuracy: 0.9362232\n",
      "\n",
      "Epoch Number: 207\n",
      "Train Loss: 0.0008333472782994735 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.38057283 Test Accuracy: 0.935725\n",
      "\n",
      "Epoch Number: 208\n",
      "Train Loss: 0.0008187590400953076 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.38099325 Test Accuracy: 0.935725\n",
      "\n",
      "Epoch Number: 209\n",
      "Train Loss: 0.0008049521249183135 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.3814406 Test Accuracy: 0.9362232\n",
      "\n",
      "Epoch Number: 210\n",
      "Train Loss: 0.0007916802561551664 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.38191074 Test Accuracy: 0.9362232\n",
      "\n",
      "Epoch Number: 211\n",
      "Train Loss: 0.0007788011856450482 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.38240153 Test Accuracy: 0.9362232\n",
      "\n",
      "Epoch Number: 212\n",
      "Train Loss: 0.0007662154095843817 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.3829116 Test Accuracy: 0.9362232\n",
      "\n",
      "Epoch Number: 213\n",
      "Train Loss: 0.0007538625193849104 Train Accuracy: 1.0\n",
      "Test Loss: 0.38343957 Test Accuracy: 0.9362232\n",
      "\n",
      "Epoch Number: 214\n",
      "Train Loss: 0.0007416940372347934 Train Accuracy: 1.0\n",
      "Test Loss: 0.38398492 Test Accuracy: 0.9362232\n",
      "\n",
      "Epoch Number: 215\n",
      "Train Loss: 0.0007296792061661357 Train Accuracy: 1.0\n",
      "Test Loss: 0.3845468 Test Accuracy: 0.9362232\n",
      "\n",
      "Epoch Number: 216\n",
      "Train Loss: 0.0007177960753125101 Train Accuracy: 1.0\n",
      "Test Loss: 0.38512433 Test Accuracy: 0.9367215\n",
      "\n",
      "Epoch Number: 217\n",
      "Train Loss: 0.0007060262115834893 Train Accuracy: 1.0\n",
      "Test Loss: 0.3857176 Test Accuracy: 0.9367215\n",
      "\n",
      "Epoch Number: 218\n",
      "Train Loss: 0.0006943566685587117 Train Accuracy: 1.0\n",
      "Test Loss: 0.38632545 Test Accuracy: 0.9367215\n",
      "\n",
      "Epoch Number: 219\n",
      "Train Loss: 0.0006827760203932859 Train Accuracy: 1.0\n",
      "Test Loss: 0.38694763 Test Accuracy: 0.9367215\n",
      "\n",
      "Epoch Number: 220\n",
      "Train Loss: 0.0006712808288483918 Train Accuracy: 1.0\n",
      "Test Loss: 0.38758373 Test Accuracy: 0.9367215\n",
      "\n",
      "Epoch Number: 221\n",
      "Train Loss: 0.0006598622694831622 Train Accuracy: 1.0\n",
      "Test Loss: 0.38823336 Test Accuracy: 0.9367215\n",
      "\n",
      "Epoch Number: 222\n",
      "Train Loss: 0.0006485197959030812 Train Accuracy: 1.0\n",
      "Test Loss: 0.3888961 Test Accuracy: 0.9367215\n",
      "\n",
      "Epoch Number: 223\n",
      "Train Loss: 0.0006372484367353561 Train Accuracy: 1.0\n",
      "Test Loss: 0.38957155 Test Accuracy: 0.9367215\n",
      "\n",
      "Epoch Number: 224\n",
      "Train Loss: 0.0006260495690297183 Train Accuracy: 1.0\n",
      "Test Loss: 0.3902598 Test Accuracy: 0.9367215\n",
      "\n",
      "Epoch Number: 225\n",
      "Train Loss: 0.0006149213456896402 Train Accuracy: 1.0\n",
      "Test Loss: 0.39096028 Test Accuracy: 0.9362232\n",
      "\n",
      "Epoch Number: 226\n",
      "Train Loss: 0.0006038654383913014 Train Accuracy: 1.0\n",
      "Test Loss: 0.39167294 Test Accuracy: 0.9362232\n",
      "\n",
      "Epoch Number: 227\n",
      "Train Loss: 0.0005928805896897532 Train Accuracy: 1.0\n",
      "Test Loss: 0.3923978 Test Accuracy: 0.9362232\n",
      "\n",
      "Epoch Number: 228\n",
      "Train Loss: 0.0005819705751093028 Train Accuracy: 1.0\n",
      "Test Loss: 0.3931346 Test Accuracy: 0.9362232\n",
      "\n",
      "Epoch Number: 229\n",
      "Train Loss: 0.0005711371570264232 Train Accuracy: 1.0\n",
      "Test Loss: 0.39388332 Test Accuracy: 0.9362232\n",
      "\n",
      "Epoch Number: 230\n",
      "Train Loss: 0.0005603803631495556 Train Accuracy: 1.0\n",
      "Test Loss: 0.39464444 Test Accuracy: 0.935725\n",
      "\n",
      "Epoch Number: 231\n",
      "Train Loss: 0.0005497033376890962 Train Accuracy: 1.0\n",
      "Test Loss: 0.3954174 Test Accuracy: 0.935725\n",
      "\n",
      "Epoch Number: 232\n",
      "Train Loss: 0.0005391083086828051 Train Accuracy: 1.0\n",
      "Test Loss: 0.3962025 Test Accuracy: 0.935725\n",
      "\n",
      "Epoch Number: 233\n",
      "Train Loss: 0.0005285999931439706 Train Accuracy: 1.0\n",
      "Test Loss: 0.39699972 Test Accuracy: 0.935725\n",
      "\n",
      "Epoch Number: 234\n",
      "Train Loss: 0.0005181774882558249 Train Accuracy: 1.0\n",
      "Test Loss: 0.39780933 Test Accuracy: 0.935725\n",
      "\n",
      "Epoch Number: 235\n",
      "Train Loss: 0.0005078478582755165 Train Accuracy: 1.0\n",
      "Test Loss: 0.39863133 Test Accuracy: 0.935725\n",
      "\n",
      "Epoch Number: 236\n",
      "Train Loss: 0.000497609365046541 Train Accuracy: 1.0\n",
      "Test Loss: 0.39946586 Test Accuracy: 0.935725\n",
      "\n",
      "Epoch Number: 237\n",
      "Train Loss: 0.0004874655870443261 Train Accuracy: 1.0\n",
      "Test Loss: 0.40031332 Test Accuracy: 0.935725\n",
      "\n",
      "Epoch Number: 238\n",
      "Train Loss: 0.0004774212549370395 Train Accuracy: 1.0\n",
      "Test Loss: 0.40117365 Test Accuracy: 0.9352267\n",
      "\n",
      "Epoch Number: 239\n",
      "Train Loss: 0.0004674753974341749 Train Accuracy: 1.0\n",
      "Test Loss: 0.40204704 Test Accuracy: 0.935725\n",
      "\n",
      "Epoch Number: 240\n",
      "Train Loss: 0.0004576308943198879 Train Accuracy: 1.0\n",
      "Test Loss: 0.40293333 Test Accuracy: 0.9352267\n",
      "\n",
      "Epoch Number: 241\n",
      "Train Loss: 0.0004478914650039084 Train Accuracy: 1.0\n",
      "Test Loss: 0.40383312 Test Accuracy: 0.93472844\n",
      "\n",
      "Epoch Number: 242\n",
      "Train Loss: 0.00043825685871487534 Train Accuracy: 1.0\n",
      "Test Loss: 0.40474612 Test Accuracy: 0.9337319\n",
      "\n",
      "Epoch Number: 243\n",
      "Train Loss: 0.0004287295363211928 Train Accuracy: 1.0\n",
      "Test Loss: 0.40567285 Test Accuracy: 0.9337319\n",
      "\n",
      "Epoch Number: 244\n",
      "Train Loss: 0.00041931199118389734 Train Accuracy: 1.0\n",
      "Test Loss: 0.4066128 Test Accuracy: 0.9337319\n",
      "\n",
      "Epoch Number: 245\n",
      "Train Loss: 0.0004100034349081298 Train Accuracy: 1.0\n",
      "Test Loss: 0.40756655 Test Accuracy: 0.9342302\n",
      "\n",
      "Epoch Number: 246\n",
      "Train Loss: 0.000400808029740328 Train Accuracy: 1.0\n",
      "Test Loss: 0.408534 Test Accuracy: 0.93472844\n",
      "\n",
      "Epoch Number: 247\n",
      "Train Loss: 0.00039172325889685205 Train Accuracy: 1.0\n",
      "Test Loss: 0.40951535 Test Accuracy: 0.93472844\n",
      "\n",
      "Epoch Number: 248\n",
      "Train Loss: 0.00038275338309874424 Train Accuracy: 1.0\n",
      "Test Loss: 0.41051057 Test Accuracy: 0.93472844\n",
      "\n",
      "Epoch Number: 249\n",
      "Train Loss: 0.00037389971720124915 Train Accuracy: 1.0\n",
      "Test Loss: 0.4115201 Test Accuracy: 0.93472844\n",
      "\n",
      "Epoch Number: 250\n",
      "Train Loss: 0.0003651603550031424 Train Accuracy: 1.0\n",
      "Test Loss: 0.4125443 Test Accuracy: 0.9352267\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch Number: 251\n",
      "Train Loss: 0.0003565376773372943 Train Accuracy: 1.0\n",
      "Test Loss: 0.4135833 Test Accuracy: 0.9352267\n",
      "\n",
      "Epoch Number: 252\n",
      "Train Loss: 0.0003480334934253845 Train Accuracy: 1.0\n",
      "Test Loss: 0.41463742 Test Accuracy: 0.9352267\n",
      "\n",
      "Epoch Number: 253\n",
      "Train Loss: 0.0003396494167359316 Train Accuracy: 1.0\n",
      "Test Loss: 0.41570726 Test Accuracy: 0.9352267\n",
      "\n",
      "Epoch Number: 254\n",
      "Train Loss: 0.0003313843925540935 Train Accuracy: 1.0\n",
      "Test Loss: 0.41679344 Test Accuracy: 0.93472844\n",
      "\n",
      "Epoch Number: 255\n",
      "Train Loss: 0.00032324064602718165 Train Accuracy: 1.0\n",
      "Test Loss: 0.4178963 Test Accuracy: 0.9352267\n",
      "\n",
      "Epoch Number: 256\n",
      "Train Loss: 0.00031521800582134485 Train Accuracy: 1.0\n",
      "Test Loss: 0.4190162 Test Accuracy: 0.93472844\n",
      "\n",
      "Epoch Number: 257\n",
      "Train Loss: 0.0003073199977859064 Train Accuracy: 1.0\n",
      "Test Loss: 0.42015436 Test Accuracy: 0.93472844\n",
      "\n",
      "Epoch Number: 258\n",
      "Train Loss: 0.00029954845147535896 Train Accuracy: 1.0\n",
      "Test Loss: 0.42131123 Test Accuracy: 0.9352267\n",
      "\n",
      "Epoch Number: 259\n",
      "Train Loss: 0.00029190532083316924 Train Accuracy: 1.0\n",
      "Test Loss: 0.42248726 Test Accuracy: 0.935725\n",
      "\n",
      "Epoch Number: 260\n",
      "Train Loss: 0.0002843899977686879 Train Accuracy: 1.0\n",
      "Test Loss: 0.42368302 Test Accuracy: 0.9352267\n",
      "\n",
      "Epoch Number: 261\n",
      "Train Loss: 0.0002770060498932415 Train Accuracy: 1.0\n",
      "Test Loss: 0.42489874 Test Accuracy: 0.9352267\n",
      "\n",
      "Epoch Number: 262\n",
      "Train Loss: 0.00026975655114941605 Train Accuracy: 1.0\n",
      "Test Loss: 0.4261356 Test Accuracy: 0.9352267\n",
      "\n",
      "Epoch Number: 263\n",
      "Train Loss: 0.0002626420347752011 Train Accuracy: 1.0\n",
      "Test Loss: 0.4273931 Test Accuracy: 0.9352267\n",
      "\n",
      "Epoch Number: 264\n",
      "Train Loss: 0.0002556652377267075 Train Accuracy: 1.0\n",
      "Test Loss: 0.4286726 Test Accuracy: 0.9352267\n",
      "\n",
      "Epoch Number: 265\n",
      "Train Loss: 0.0002488270239491488 Train Accuracy: 1.0\n",
      "Test Loss: 0.42997336 Test Accuracy: 0.9352267\n",
      "\n",
      "Epoch Number: 266\n",
      "Train Loss: 0.00024212878793462064 Train Accuracy: 1.0\n",
      "Test Loss: 0.4312961 Test Accuracy: 0.93472844\n",
      "\n",
      "Epoch Number: 267\n",
      "Train Loss: 0.00023557050247303505 Train Accuracy: 1.0\n",
      "Test Loss: 0.43264046 Test Accuracy: 0.93472844\n",
      "\n",
      "Epoch Number: 268\n",
      "Train Loss: 0.000229153466771344 Train Accuracy: 1.0\n",
      "Test Loss: 0.43400633 Test Accuracy: 0.9342302\n",
      "\n",
      "Epoch Number: 269\n",
      "Train Loss: 0.00022287752135650396 Train Accuracy: 1.0\n",
      "Test Loss: 0.43539396 Test Accuracy: 0.93472844\n",
      "\n",
      "Epoch Number: 270\n",
      "Train Loss: 0.00021674406400974203 Train Accuracy: 1.0\n",
      "Test Loss: 0.43680328 Test Accuracy: 0.93472844\n",
      "\n",
      "Epoch Number: 271\n",
      "Train Loss: 0.0002107478593307075 Train Accuracy: 1.0\n",
      "Test Loss: 0.43823338 Test Accuracy: 0.93472844\n",
      "\n",
      "Epoch Number: 272\n",
      "Train Loss: 0.00020489141274058605 Train Accuracy: 1.0\n",
      "Test Loss: 0.43968424 Test Accuracy: 0.9352267\n",
      "\n",
      "Epoch Number: 273\n",
      "Train Loss: 0.00019917354314214844 Train Accuracy: 1.0\n",
      "Test Loss: 0.4411549 Test Accuracy: 0.9352267\n",
      "\n",
      "Epoch Number: 274\n",
      "Train Loss: 0.00019359330177045594 Train Accuracy: 1.0\n",
      "Test Loss: 0.44264516 Test Accuracy: 0.9367215\n",
      "\n",
      "Epoch Number: 275\n",
      "Train Loss: 0.00018814832334126 Train Accuracy: 1.0\n",
      "Test Loss: 0.4441542 Test Accuracy: 0.9367215\n",
      "\n",
      "Epoch Number: 276\n",
      "Train Loss: 0.00018283885997147554 Train Accuracy: 1.0\n",
      "Test Loss: 0.4456814 Test Accuracy: 0.9367215\n",
      "\n",
      "Epoch Number: 277\n",
      "Train Loss: 0.00017766496358951234 Train Accuracy: 1.0\n",
      "Test Loss: 0.4472254 Test Accuracy: 0.9367215\n",
      "\n",
      "Epoch Number: 278\n",
      "Train Loss: 0.00017262536239784771 Train Accuracy: 1.0\n",
      "Test Loss: 0.44878626 Test Accuracy: 0.9367215\n",
      "\n",
      "Epoch Number: 279\n",
      "Train Loss: 0.00016772304865896818 Train Accuracy: 1.0\n",
      "Test Loss: 0.4503633 Test Accuracy: 0.9367215\n",
      "\n",
      "Epoch Number: 280\n",
      "Train Loss: 0.00016296197600863645 Train Accuracy: 1.0\n",
      "Test Loss: 0.451956 Test Accuracy: 0.93721974\n",
      "\n",
      "Epoch Number: 281\n",
      "Train Loss: 0.00015834992364158995 Train Accuracy: 1.0\n",
      "Test Loss: 0.45356566 Test Accuracy: 0.9367215\n",
      "\n",
      "Epoch Number: 282\n",
      "Train Loss: 0.00015390685763272253 Train Accuracy: 1.0\n",
      "Test Loss: 0.45519423 Test Accuracy: 0.9367215\n",
      "\n",
      "Epoch Number: 283\n",
      "Train Loss: 0.0001496767374241967 Train Accuracy: 1.0\n",
      "Test Loss: 0.4568483 Test Accuracy: 0.9367215\n",
      "\n",
      "Epoch Number: 284\n",
      "Train Loss: 0.00014578019630017192 Train Accuracy: 1.0\n",
      "Test Loss: 0.45854405 Test Accuracy: 0.9362232\n",
      "\n",
      "Epoch Number: 285\n",
      "Train Loss: 0.00014262170305001957 Train Accuracy: 1.0\n",
      "Test Loss: 0.46033803 Test Accuracy: 0.9362232\n",
      "\n",
      "Epoch Number: 286\n",
      "Train Loss: 0.00014241321601630636 Train Accuracy: 1.0\n",
      "Test Loss: 0.46250543 Test Accuracy: 0.9362232\n",
      "\n",
      "Epoch Number: 287\n",
      "Train Loss: 0.00039112994681377196 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.47080484 Test Accuracy: 0.937718\n",
      "\n",
      "Epoch Number: 288\n",
      "Train Loss: 0.002160546216233243 Train Accuracy: 0.9993150691463523\n",
      "Test Loss: 0.4985566 Test Accuracy: 0.9332337\n",
      "\n",
      "Epoch Number: 289\n",
      "Train Loss: 0.0015827084215531725 Train Accuracy: 0.9997260276585409\n",
      "Test Loss: 0.460808 Test Accuracy: 0.9342302\n",
      "\n",
      "Epoch Number: 290\n",
      "Train Loss: 0.0013418768471824914 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.4653932 Test Accuracy: 0.93472844\n",
      "\n",
      "Epoch Number: 291\n",
      "Train Loss: 0.00042524092328078463 Train Accuracy: 0.9998630138292705\n",
      "Test Loss: 0.46051893 Test Accuracy: 0.93472844\n",
      "\n",
      "Epoch Number: 292\n",
      "Train Loss: 0.0002157161274326053 Train Accuracy: 1.0\n",
      "Test Loss: 0.45942155 Test Accuracy: 0.935725\n",
      "\n",
      "Epoch Number: 293\n",
      "Train Loss: 0.00018626744945135688 Train Accuracy: 1.0\n",
      "Test Loss: 0.45938873 Test Accuracy: 0.935725\n",
      "\n",
      "Epoch Number: 294\n",
      "Train Loss: 0.000173948956017577 Train Accuracy: 1.0\n",
      "Test Loss: 0.4597626 Test Accuracy: 0.935725\n",
      "\n",
      "Epoch Number: 295\n",
      "Train Loss: 0.00016522929556779436 Train Accuracy: 1.0\n",
      "Test Loss: 0.4602842 Test Accuracy: 0.9352267\n",
      "\n",
      "Epoch Number: 296\n",
      "Train Loss: 0.00015823588222552295 Train Accuracy: 1.0\n",
      "Test Loss: 0.46087208 Test Accuracy: 0.9352267\n",
      "\n",
      "Epoch Number: 297\n",
      "Train Loss: 0.00015232920922655517 Train Accuracy: 1.0\n",
      "Test Loss: 0.4614972 Test Accuracy: 0.9352267\n",
      "\n",
      "Epoch Number: 298\n",
      "Train Loss: 0.00014718409047792156 Train Accuracy: 1.0\n",
      "Test Loss: 0.46214733 Test Accuracy: 0.9352267\n",
      "\n",
      "Epoch Number: 299\n",
      "Train Loss: 0.00014260701286667889 Train Accuracy: 1.0\n",
      "Test Loss: 0.46281824 Test Accuracy: 0.9352267\n",
      "\n",
      "Maximum Test accuracy at compressed model size(including early stopping): 0.9412058 at Epoch: 146\n",
      "Final Test Accuracy: 0.9352267\n",
      "\n",
      "\n",
      "Non-Zeros: 1932 Model Size: 7.546875 KB hasSparse: False\n",
      "\n",
      "The Model Directory: usps10\\FastGRNNResults/23_51_17_15_03_19\n",
      "\n"
     ]
    }
   ],
   "source": [
    "FastCellTrainer.train(batchSize, totalEpochs, sess, Xtrain, Xtest,\n",
    "                      Ytrain, Ytest, decayStep, decayRate, dataDir, currDir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Quantization\n",
    "\n",
    "Byte Quantization for the trained FastModels, to reduce the model size by 4x. If one uses piece-wise linear approximations for non-linearities like quantTanh for tanh and quantSigm for Sigmoid, they can benefit greatly from pure integer arithmetic after model quantization during prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bg.npy has max: 4.9833384 min: -0.6077357\n",
      "Bh.npy has max: 2.8973198 min: -0.16004847\n",
      "FC.npy has max: 4.9540076 min: -5.963999\n",
      "FCbias.npy has max: 2.540496 min: -1.7358814\n",
      "U.npy has max: 2.2965062 min: -2.670992\n",
      "W.npy has max: 1.3919494 min: -1.2454427\n",
      "\n",
      "\n",
      "Quantized Model Dir: usps10\\FastGRNNResults/23_51_17_15_03_19\\QuantizedFastModel\n"
     ]
    }
   ],
   "source": [
    "#Model quantization\n",
    "model_dir = currDir #you will see model dir printed at the end of trianing, use that here or use the currDir\n",
    "\n",
    "import quantizeFastModels\n",
    "quantizeFastModels.quantizeFastModels(model_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
